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Abstract 

Increasing numbers of research studies test interventions for clinicians in addition to or instead of interventions for 
patients. Although previous studies have enumerated barriers to patient enrolment in clinical trials, corresponding 
barriers have not been identified for enrolling clinicians as subjects. We propose a framework of metrics for 
evidence-based estimation of time and resources required for recruiting clinicians as research participants, and 
present an example from a federally funded study. Our framework proposes metrics for tracking five steps in the 
recruitment process: gaining entry into facilities, obtaining accurate eligibility and contact information, reaching 
busy clinicians, assessing willingness to participate, and scheduling participants for data collection. We analyzed 
recruitment records from a qualitative study exploring performance feedback at US Department of Veterans Affairs 
Medical Centers (VAMCs); five recruiters sought to reach two clinicians at 16 facilities for a one-hour interview. 
Objective metrics were calculable for all five steps; metric values varied considerably across facilities. Obtaining 
accurate contact information slowed down recruiting the most. We conclude that successfully recruiting even small 
numbers of employees requires considerable resourcefulness and more calendar time than anticipated. Our 
proposed framework provides an empirical basis for estimating research-recruitment timelines, planning 
subject-recruitment strategies, and assessing the research accessibility of clinical sites. 



Background 

Federal research funding has seen an increase in re- 
search studies whose interventions for improving health- 
care quality are aimed at clinicians in addition to or in 
lieu of patients; for example, the number of projects 
awarded by the National Institutes of Health focusing on 
providers has tripled from 2010 to 2012 [1]. Studying cli- 
nicians as the population of interest is certainly nothing 
new; for example, the Nurses' Health Study [2], ori- 
ginally designed to investigate the potential long-term 
consequences of the use of oral contraceptives, has suc- 
cessfully followed registered nurses prospectively for 
over 35 years, monitoring various health-related and 
quality of life topics every two years. In the context of 
implementation research, however, clinicians serve not 
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only as sources of data and/or the objects of inter- 
vention, but often as stakeholders and indispensable 
resources to ensure the success of the intervention's im- 
plementation. Consequently, successfully recruiting cli- 
nicians in implementation studies becomes not only a 
matter of scientific importance, but of implementation 
success as well. 

As clinician time becomes more precious, it becomes 
increasingly important for researchers to realistically 
gauge their access to this already stressed, vulnerable re- 
search population. Unlike the literature on patient re- 
cruitment, which has a rich, diverse knowledgebase of 
hundreds of studies, research on recruiting clinicians as 
research subjects is fragmented and scarce. For example, 
common strategies for maximizing recruitment of clini- 
cians include minimizing respondent burden, sizeable 
incentives, capitalizing on existing relationships, and 
personal contact with target subjects; such strategies are 
drawn from papers examining different subject popula- 
tions, such as individual physicians, individual nurses, 
and entire medical practices [3-5]. Consistent with some 



© 201 3 Hysong et at; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative 
Commons Attribution License (http://creativecommons.Org/licenses/by/2.0), which permits unrestricted use, distribution, and 
reproduction in any medium, provided the original work is properly cited. 



Hysong et al. Implementation Science 2013, 8:125 
http://www.implementationscience.eom/content/8/1/125 



Page 2 of 7 



of these strategies, Solberg and colleagues [6] proposed 
seven factors researchers must mind to maximize parti- 
cipation by entire medical groups, including longstanding 
relationships with the target clinicians, an established re- 
putation, and rewards; however, achieving the 100% par- 
ticipation rate they reported required herculean efforts 
not often included in the timelines or budgets of research 
proposals. Other factors that have been noted to impact 
clinician recruitment include less obvious matters, such as 
institutional affiliation [6], clinical setting [7], and delays 
in Institutional Review Board (IRB) approvals [8]. 

The aforementioned factors, while important and ef- 
fective, provide uncoordinated guidance to help design a 
successful recruitment campaign, and even less help in 
estimating the time and resources required. Recently, 
Broyles and colleagues noted that most published studies 
using clinicians as subjects (especially nurses) 'provide 
little information regarding recruitment strategies, re- 
cruitment rates, participation rates, and representative- 
ness rates' [3]. It is not surprising, then, that researchers 
often design their clinician recruitment strategy heuris- 
tically, or that reviewers often rely on personal expe- 
rience to judge the adequacy of a research proposal's 
recruitment plan. This could lead to considerable devia- 
tions from originally proposed timelines and budgets, 
potentially resulting in wasted effort and research 
dollars. 

Recruiting clinicians as participants often takes far lon- 
ger than anticipated, adversely affecting timelines, bud- 
gets, and, potentially, research findings; barriers exist at 
all points in recruitment, including: gaining entry into 
facilities, obtaining accurate eligibility and contact infor- 
mation, reaching busy clinicians, persuading eligible can- 
didates to participate (without coercion), and scheduling 
willing participants for data collection. 

Although previous research of patient recruitment has 
identified numerous barriers to patient enrolment in 



clinical trials, including physician time restrictions and 
clinical trial awareness [9-11], corresponding barriers 
have not been identified for enrolling clinicians as sub- 
jects. More importantly, we are unaware of any existing 
metrics to guide investigators in research planning. Con- 
sequently, our objective is to propose a framework of 
metrics for estimating time and resources required for 
recruiting clinicians as research participants in clinical 
studies; as an illustrative example, we present initial esti- 
mates from a federally funded study recruiting clinicians 
at 16 sites. With this framework, we aim to provide an 
evidence-based means of estimating and evaluating the 
feasibility of clinician-recruitment strategies in funded 
research. 

Method 

Proposed framework 

Normally, recruitment is a single line in a timetable for 
a research project. A more detailed, task-based approach 
is warranted to accurately estimate recruitment time and 
resources needed. The process map methodology [12] is 
commonly used in work analysis as well as in Lean pro- 
cesses to identify both the universe of activities in a 
work process and critical barriers to efficient and effec- 
tive outcomes. Thus, to accomplish our goal, we created 
a process map of our recruitment approach, and iden- 
tified five basic activities for which metrics could be 
devised and time estimates calculated: gaining entry; 
obtaining accurate records; reaching participants; asses- 
sing willingness to participate; and scheduling partici- 
pants for the desired procedure. Table 1 describes the 
specific metrics we formulated for each of these recruit- 
ment tasks. Although recruitment tactics may vary from 
project to project, we propose that these five basic acti- 
vities are applicable to recruitment approaches for a 
wide variety of research designs. 



Table 1 Metrics quantifying efforts to recruit clinicians for a one-hour telephone interview 



Step 


Metric 


Mean or % 


SD 


Min-max n 


Gaining entry 


1. Number of contact attempts to site needed to establish 
authorization to recruit at a given site. 


6.4 


8.0 


1-27 16 


Obtaining accurate records (Figure 1) 


2. Percent of potential participants assumed eligible from 
employee database queries who were actually ineligible 
to participate ('dead ends'). 


67% (148/221) 






Reaching participants 


3. Number of contact attempts to a potential participant 
prior to receiving an initial response. 


3.6 


3.5 


1-18 68 




4. Cycle time in calendar days from initial contact attempt 
to initial participant response. 


9.5 


9.9 


0-50 68 


Assessing willingness to participate 
(Figure 1) 


5. Percent of potential participants who declined (of those 
who responded to recruiter inquiry and were not lost to 
contact or ineligible). 


53% (21/40) 






Scheduling participants 


6. Cycle time in calendar days from initial contact to interview 
completion for those agreeing to participate. 


24.4 


16.4 


6-62 19 
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Example study 

To demonstrate the utility of this framework, we calcu- 
lated actual metric values for each of the five tasks for a 
federally funded study; by calculating actual values, we 
not only demonstrate the utility of the framework using 
real data but also provide initial estimates for future re- 
searchers attempting to implement similar recruitment 
plans. Specific methods for the study have been pub- 
lished elsewhere in this journal [13]; briefly, its objective 
was to identify differences in the audit and feedback 
practices of high- versus low- and moderately perfor- 
ming outpatient clinics. Among other participants, we 
attempted to recruit one physician and one nurse at 
each of 16 sites for a one-hour interview, the content of 
which included questions about the types of quality/clin- 
ical performance information they receive and actively 
seek out, opinions and attitudes about the utility of said 
data as a form of feedback, and how they use this 
information. 

Participants and site selection 

On the basis of filtered lists of eligible candidates, we 
targeted 221 primary care registered nurses and physi- 
cians at 16 geographically dispersed Veterans Affairs 
Medical Centers (VAMCs) for inclusion in this study, 
for a desired final sample of 32 clinician participants. 
Only full-time clinicians with at least three years in their 
current position were eligible. 

Sites were selected using a purposive stratified ap- 
proach based on their scores on a profile of 15 out- 
patient clinical performance measures extracted from 
VAs External Peer Review Program (EPRP). We selected 
four high-performing, four low-performing, four consist- 
ently moderately performing (i.e., those with moderate 
average performance and the lowest variability across 
measures), and four highly variable facilities (i.e., those 
with moderate average performance and the highest 
variability across measures). For the present paper, where 
we explored clinician recruitment cycle times, the re- 
search team and all analyses conducted were blind to 
facility type; however, diverse facility performance levels 
ensured a broad representation of sites (see Additional 
file 1 for study site descriptive statistics). 

Participant recruitment 

Prospective participants received an email inviting them 
to participate in the study and requesting a preferred 
contact email, phone number, and a time when a re- 
search team member can review the study information 
form with them and obtain informed consent. Research 
team members phoned the participants at their re- 
quested appointment time to review the consent form 
(which they received in advance of the call) and proce- 
dures with the participant, answer any questions, and 



schedule the interview upon agreeing to participate. Ex- 
amples of the recruitment materials are presented in 
Additional file 2. No monetary incentives were provided 
for participating in the study (as are often offered in 
interview studies), as federal regulations prevented us 
from providing such incentives to federal (in this case 
VA) employees. 

Measures 

We calculated cycle times for accomplishing each of the 
five recruitment tasks described earlier, using recruit- 
ment records from a qualitative study exploring perfor- 
mance feedback at VAMCs [14] (Table 1). 

Data sources 

Potential participant lists 

Recruiters secured filtered lists of eligible candidates 
from the VAs Personnel Accounting Integrated Database 
(PAID); this is VAs national employee database, which 
includes employment site, job title, length of employment, 
and full-time or part-time status (study inclusion criteria). 
These were used as start values for assessing task #2, 
obtaining accurate eligibility and contact information. 

Contact/communications records 

Five recruiters tasked to schedule two clinicians per fa- 
cility for a one-hour telephone interview maintained a 
database of communications to sites and potential par- 
ticipants. The database was originally designed to track 
communications to potential participants for IRB audi- 
ting and documentation purposes. We queried this data- 
base to calculate number of contact attempts to site 
personnel and cycle times from initial contact attempt 
to interview completion. 

Results 

Table 1 summarizes our observations. Table 2 provides 
descriptive characteristics of the clinicians interviewed. 
Objective metrics were calculable for all five tasks; most 
striking, however, was the variability in the values for 
each metric. For example, the standard deviations for 
both Reaching Participants metrics were almost equal in 



Table 2 Characteristics of qualitative study interviewees 



Characteristics 


MD (n = 


12) 


RN (n = 


12) 




W 


% 


N 


% 


Gender 










• Male 


3 


25 


1 


8.3 


• Female 


9 


75 


11 


91.7 


Full time status 


12 


100 


12 


100 




M 


SD 


M 


SD 


Length of time at VA 


8.7 


5.3 


11.7 


6.9 
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magnitude to the metrics mean, indicating wide varia- 
bility in cycle times and contact attempts observed. In 
addition, considerable variation existed across facilities 
in the accuracy of initial filtered lists obtained from the 
employee database, resulting in high percentages of in- 
eligible prospects even after filtering lists for eligibility 
criteria (Figure 1). Although it was impossible to track 
cycle time for obtaining an accurate candidate list for a 
given facility with our available data sources, we inter- 
viewed the study's recruiters to create a process map dis- 
playing all needed steps and decision points to obtain 
accurate contact information. Figure 2 displays a gener- 
alized map of our process for obtaining accurate contact 
information (Step 2 in our framework) and identifies 
all nonvalue-added steps. As shown, obtaining accurate 
contact information was much more complicated than 
expected. Depending on the site and its data's accuracy 
and completeness, the process could be as short as seven 
steps (bold arrows indicate this critical path) or drama- 
tically complicated by non-value added (black and grey 
boxes) actions, approvals, and error loops. Recruiters 
were thus forced to consult multiple sources (e.g., enter- 
prise employee email directories, facility websites, clinic 
cold calls) to confirm eligibility (when possible) before 
contact, considerably delaying initial contact with the 
participant. Limited avenues for contact {e.g., no direct 
phone line to a clinician office) often prevented recrui- 
ters from reaching clinicians expeditiously (Table 1). 
Patient-care duties often interfered with clinicians' ability 
to respond to recruiter communications. Lack of time to 
participate in an hour-long interview was the modal rea- 
son for declining our invitations. 

Discussion 

This article proposes a framework for estimating time 
and resources required for recruiting clinicians as re- 
search participants in clinical studies, thereby providing 
more evidence-based means of estimating and evalua- 
ting the feasibility of clinician-recruitment strategies in 



funded research. Using a currently ongoing federally 
funded study as an example, we demonstrate our frame- 
work's applicability to the clinician-subject population; 
our proposed metrics were calculable and obtainable; 
and, in our case, we found wide variability across study 
sites in the effort required to recruit clinicians. We are 
unaware of prior health services research studies that ac- 
complish this objective: a PubMed search of studies 
examining human-subject research recruitment found 
no studies examining clinicians as subjects. 

One of the most striking observations in the illustra- 
tive study was the high variability in cycle times, possibly 
a function of the facilities' infrastructure and orga- 
nizational idiosyncrasies. Because we targeted only two 
people at each facility, we could not calculate within-site 
statistics to confirm this. However, based on the content 
of the interviews eventually conducted for the qualitative 
study, we observed considerable variability in how these 
16 sites operated, which could have certainly accounted 
for the observed cycle time variability. These observa- 
tions are therefore a reminder to researchers to consider 
in their estimates of recruitment time the types of orga- 
nizational systems that could facilitate or hinder the re- 
cruitment process. 

Our findings highlight the difficulty of accessing clini- 
cians for research purposes and the considerable barriers 
that exist even before the first contact attempt is made. 
At minimum, our study is a reminder to research teams 
to map their recruitment process, as well as to identify, 
anticipate, and prepare for potential bottlenecks. For ex- 
ample, site entry (Step 1) can often be a considerable 
barrier, especially when no a priori relationships exist be- 
tween the site and the research team. Consistent with 
existing models of implementation [14-16], relationships 
with stakeholders that would normally be built as part of 
the implementation process could be leveraged to help 
gain access to both the site and to clinicians, thereby 
shortening cycle time. Similarly, as illustrated in Figure 2, 
up-to-date employee records (Step 2) could considerably 



Considered for inclusion (221) 



Invited to participate (104) 



Interviewed (19) 




Excluded due to ineligibility (1 1 7) 




Modal reasons for ineligibility: 

Not working in primary care 
Not working at the study facility 
• Not enough years in role* 
Not full-time in role* 




Deemed ineligible after contact (31) 








• No longer working at VA 


No response (24) 
Lost/Ceosed contact (9) 




'Determined after contact with subject 







Declined (21) 



Figure 1 Outcomes for potential participants identified using VA PAID, filtered to eligibility requirements. 
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Figure 2 Generalized process map for obtaining accurate contact information for potential subjects. 
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shorten steps required, and therefore cycle times to ini- 
tial participant contact, thus reducing wasted effort. Al- 
though the freshness of employee records is beyond the 
investigators' control, facility stakeholders could advise 
investigators on more appropriate sources of contact in- 
formation, and possibly suggest alternative means of rea- 
ching participants (e.g., approaching clinicians en masse 
as part of a regularly scheduled staff meeting). 

Additionally, our study also highlights the utility of 
tracking participant communications throughout the re- 
cruitment and data collection process. Although not ini- 
tially intended for this purpose, the example study's 
communications tracker made it possible to generate es- 
timates of time and resources for recruiting. Tracking 
participant communications is a practice that many 
studies already employ, often for legal or regulatory pur- 
poses. A few minor edits to a study's participant com- 
munications tracking system (e.g., adding database fields, 



data entry, and/or reporting tools) could greatly facilitate 
participant recruitment tracking and help easily generate 
the metrics proposed in this study. 

Our findings show initial estimates for one-hour inter- 
views for a qualitative study; quantitative studies em- 
ploying different data collection techniques such as 
surveys or medical chart abstraction are likely to exhibit 
different time estimates because of logistical concerns 
unique to those methods (e.g., online accessibility). How- 
ever, the metrics proposed here are nevertheless viable 
and calculable for these types of methods because the 
initial steps of gaining entry, securing contact information, 
contact attempts, etc., are common to all these tech- 
niques. Just as analyses of late response and nonresponse 
are becoming the preferred way of assessing the quality of 
survey-based data [17,18], the proposed framework pro- 
vides a way to estimate feasibility of work proposed 
in grant applications. As new, less traditional research 
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designs emerge, perhaps new steps and metrics could 
be added to the framework; we consider our framework a 
good starting point to which future researchers can add. 

Limitations 

The tools used to generate the estimates herein were not 
originally designed for this purpose; better estimates 
could be calculated had recruitment considerations been 
designed a priori into the example study. In addition, 
only recruitment challenges related to the participant-to- 
recruiter interaction were examined. Other types of ex- 
ternally imposed delays were excluded (e.g., IRB-related 
barriers [8]; shifts in organizational priorities). The poten- 
tial for these to adversely impact recruiting should not be 
discounted. 

Additionally, it is possible that the subject of our inter- 
views may have been somewhat intimidating to partici- 
pants, thereby potentially affecting recruitment success. 
However, the content was not deemed sensitive by our 
IRB; no one declined to participate due to the topic of 
our interviews, and only one participant requested not 
to be audio recorded. We therefore believe that the topic 
of our interview had minimal, if any impact on the cycle 
times observed in the example study. 

More broadly, the framework is currently of limited 
value in that these metrics are not currently monitored 
or reported with currently conducted research; thus 
there are no existing empirical estimates of what can be 
considered reasonable or customary recruitment times 
for different types of designs (as alluded to by Broyles 
and colleagues) [3]. We caution that the values reported 
here are based on a small number of target subjects and 
may not be suitable as stable estimates of recruitment 
time; as more studies begin to monitor and report these 
metrics, empirical estimates can be calculated and bench- 
marks created for evaluating feasibility. In the meantime, 
our framework can be used to help guide thoughtfully de- 
signed recruitment campaigns in research proposals. 

Conclusion 

Recruiting clinicians as subjects is a laborious endeavour 
with considerable potential impact to the implementa- 
tion success of an intervention. Successfully recruiting 
even small numbers of employees requires considerable 
personnel, resourcefulness, perseverance, and more cal- 
endar time than experience and heuristics would predict. 
Our proposed framework and metrics provide an empi- 
rical basis for estimating research recruitment timelines, 
planning subject-recruitment strategies, and assessing 
the research accessibility of clinical sites. We encourage 
future researchers to report metrics such as those pro- 
posed here in their research findings so that more stable 
estimates of recruitment time for different study designs 
can be formulated. 
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